{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Train a GPT-2 Text-Generating Model w/ GPU",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/qiujz/GPT2-Chinese/blob/master/Copy_of_Train_a_GPT_2_Text_Generating_Model_w_GPU.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7LoMj4GA4n_",
        "colab_type": "text"
      },
      "source": [
        "#  免费用 Google GPU 来 finetune GPT-2\n",
        "\n",
        "英文来自： [Max Woolf](http://minimaxir.com)\n",
        "\n",
        "中文修改：Andy\n",
        "\n",
        "最近 GPT2 放出了 345M 的中型模型（之前是 117M 模型），于是又引起一波 Finetune 大潮，于是自己也就顺便抽出时间拿这个 345M 模型 finetune 了几个玩。\n",
        "\n",
        "这部分主要是教大家用 `gpt-2-simple` 库，并且蹭 Google Colab 免费 GPU 来对 GPT2 模型进行 finetune，这篇 Notebook 业主要来自这个库的作者。\n",
        "\n",
        "\n",
        " `gpt-2-simple` 库 github 地址：https://github.com/minimaxir/gpt-2-simple\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WjJs-kTnbyLx",
        "colab_type": "text"
      },
      "source": [
        "#### 开始前准备\n",
        "\n",
        "1. 首先，要想薅 Google 羊毛，就得现有个 Google 账号，[注册](https://accounts.google.com/signup);\n",
        "2. 然后把这个 Notebook 保存到自己 Google Drive 里去，便于运行和修改。(File -> Save a Copy in Drive)\n",
        "3. 之后，同样的既然要薅 Google 羊毛，就还得用他家产品，所以保证用 Chrome 浏览器；\n",
        "4. 最后，安装环境和 import 各种需要包，运行下面的 Cell 就好了:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KBkpRgBCBS2_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install -q gpt_2_simple\n",
        "import gpt_2_simple as gpt2\n",
        "from datetime import datetime\n",
        "from google.colab import files"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bj2IJLHP3KwE",
        "colab_type": "text"
      },
      "source": [
        "## 查看 GPU\n",
        "\n",
        "既然要薅羊毛，那就要薅个清楚，先来看看 GPU 信息吧。\n",
        "\n",
        "现在 Colab 更新过一次，已经再使用英伟达 T4 GPU了，比之前的 K80 要快些，而且内存也要大些，有 16 G。这么大的内存 finetune 的时候拿大模型来也都没问题了，而且可以生成更多文本。\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sUmTooTW3osf",
        "colab_type": "code",
        "outputId": "7d2fa7e3-ab5d-4a9d-dbde-0d47c8f68dc2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        }
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Wed May 15 12:28:39 2019       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 418.56       Driver Version: 410.79       CUDA Version: 10.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   38C    P8    15W /  70W |      0MiB / 15079MiB |      0%      Default |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                       GPU Memory |\n",
            "|  GPU       PID   Type   Process name                             Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0wXB05bPDYxS",
        "colab_type": "text"
      },
      "source": [
        "## 下载 GPT-2 模型\n",
        "\n",
        "首先，finetune 的第一步，先把预训练模型下载下来，就像做菜要先备好材料。\n",
        "\n",
        "正如开头提到的，现在放出了两个 GPT-2 模型， \n",
        "* `117M` : 小模型，大概有 500MB 大小；\n",
        "* `345M`: 最近放出的中型模型，1.5 GB.\n",
        "\n",
        "之后 finetune 的时候，只需要修改模型名字 `model_name` 参数就可以选择模型。大模型有更多的知识，但同时也要更长时间来进行 finetune 和生成。\n",
        "\n",
        "下面的 cell 就是用来下载预训练模型文件的，会保持在 Colab 的 `/models/<model_name>`下，点左边栏上面的 `Files` 也可以看到。\n",
        "\n",
        "模型不是永久保存的，下次 finetune 的时候还要重新下载。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8wSlgXoDPCR",
        "colab_type": "code",
        "outputId": "a269ca79-4a89-4555-e8a0-03ceae248344",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        }
      },
      "source": [
        "gpt2.download_gpt2(model_name=\"345M\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.00kit [00:00, 641kit/s]                                                      \n",
            "Fetching encoder.json: 1.04Mit [00:00, 48.7Mit/s]                                                   \n",
            "Fetching hparams.json: 1.00kit [00:00, 620kit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 1.42Git [00:26, 53.9Mit/s]                                 \n",
            "Fetching model.ckpt.index: 11.0kit [00:00, 2.89Mit/s]                                               \n",
            "Fetching model.ckpt.meta: 927kit [00:00, 46.6Mit/s]                                                 \n",
            "Fetching vocab.bpe: 457kit [00:00, 36.6Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N8KXuKWzQSsN",
        "colab_type": "text"
      },
      "source": [
        "## 装载 Google Drive （类似云盘）\n",
        "\n",
        "想输入训练数据，然后保存模型的最好方法，就是把 Google Drive 给装载进来，之后把训练数据放在上面，保存也直接放上面\n",
        "\n",
        "\n",
        "下面这个 cell 就是装载个人 Google Driver 的命令。会出现一个链接和提示框，点击链接获得授权码，粘贴到提示框里去。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "puq4iC6vUAHc",
        "colab_type": "code",
        "outputId": "d1e74e69-5c23-4eba-d951-c43b9f6298d5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "gpt2.mount_gdrive()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BT__brhBCvJu",
        "colab_type": "text"
      },
      "source": [
        "## 上传训练文本\n",
        "\n",
        "也是在左边的侧栏里面，直接点击`Files` 里面的上传 `UPLOAD` 就好了。\n",
        "\n",
        "![alt text](https://i.imgur.com/TGcZT4h.png)\n",
        "\n",
        "之后也可以传任意自己想 finetune 的文本，这里我们用老友记十季的剧本来作为训练数据。直接上传的话推荐小于 10M 的数据。\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6OFnPCLADfll",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "file_name = \"friends.txt\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeeSKtNWUedE",
        "colab_type": "text"
      },
      "source": [
        "如果大于 10M 的文本，那就推荐先上传到 google drive (根目录) 里然后再拷过来。如果报错，再重新运行装载命令。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Z6okFD8VKtS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gpt2.copy_file_from_gdrive(file_name)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdpZQXknFNY3",
        "colab_type": "text"
      },
      "source": [
        "## Finetune GPT-2\n",
        "\n",
        "于是准备好模型和数据，就要开始今天的主菜了，finetune GPT2. \n",
        "\n",
        "下面的代码会运行一个 Tensorflow Sessioin，读取训练设定，然后按照设定训练。`steps` 用来指定 finetune 多少步，如果设成 -1 的话就表示不停 finetune 下去。\n",
        "\n",
        "模型文件会保存在`/checkpoint/run1` 里面，会根据`save_every`来保存一定步数文件，结束训练的时候也会保存。\n",
        "\n",
        "训练结束需要大概半个小时，一定要记得保存模型。\n",
        "\n",
        "\n",
        "\n",
        "**重要提示**：如果想要重复运行下面 cell 的话，需要重启服务，`Runtime/Restart Runtime`.\n",
        "\n",
        "gpt2.finetune 训练参数介绍：\n",
        "\n",
        "\n",
        "*  **`restore_from`**:  `fresh` 是指从 GPT2 原模型开始, 而 `latest`是从之前 finetune 保存的模型继续训练.\n",
        "* **`sample_every`**: 每多少步输出样本，看看训练效果\n",
        "* **`print_every`**: 每多少步打印训练的一些参数，从左到右，步数、时间，loss，平均loss\n",
        "* **`learning_rate`**: 学习率 (默认 `1e-4`, 如果数据小于1MB的话可以调低到 `1e-5`)\n",
        "*  **`run_name`**: 运行的时候，保存模型到`checkpoint`下哪个子文件，默认 `run1`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aeXshJM-Cuaf",
        "colab_type": "code",
        "outputId": "22335fe4-0e05-4612-9b1c-239dc940ce5e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 5446
        }
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.finetune(sess,\n",
        "              dataset=file_name,\n",
        "              model_name='345M',\n",
        "              steps=1000,\n",
        "              restore_from='fresh',\n",
        "              print_every=10,\n",
        "              sample_every=200,\n",
        "              save_every=500\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:51: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:53: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.random.categorical instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n",
            "Loading checkpoint models/345M/model.ckpt\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n",
            "INFO:tensorflow:Restoring parameters from models/345M/model.ckpt\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r  0%|          | 0/1 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading dataset...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r100%|██████████| 1/1 [00:07<00:00,  7.86s/it]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "dataset has 1464638 tokens\n",
            "Training...\n",
            "[10 | 23.14] loss=2.54 avg=2.54\n",
            "[20 | 38.13] loss=2.74 avg=2.64\n",
            "[30 | 53.24] loss=2.31 avg=2.53\n",
            "[40 | 68.47] loss=2.62 avg=2.55\n",
            "[50 | 83.82] loss=2.55 avg=2.55\n",
            "[60 | 99.25] loss=2.29 avg=2.51\n",
            "[70 | 114.81] loss=2.28 avg=2.47\n",
            "[80 | 130.46] loss=2.31 avg=2.45\n",
            "[90 | 146.22] loss=2.56 avg=2.46\n",
            "[100 | 162.06] loss=2.74 avg=2.49\n",
            "[110 | 177.97] loss=2.46 avg=2.49\n",
            "[120 | 193.95] loss=2.13 avg=2.46\n",
            "[130 | 209.99] loss=2.58 avg=2.47\n",
            "[140 | 226.06] loss=2.43 avg=2.47\n",
            "[150 | 242.11] loss=2.49 avg=2.47\n",
            "[160 | 258.17] loss=2.75 avg=2.49\n",
            "[170 | 274.26] loss=2.16 avg=2.47\n",
            "[180 | 290.37] loss=2.50 avg=2.47\n",
            "[190 | 306.50] loss=1.98 avg=2.44\n",
            "[200 | 322.64] loss=2.31 avg=2.43\n",
            "======== SAMPLE 1 ========\n",
            "\n",
            "Skipper: But I don''t think it was that bad!\n",
            "Joey: Hey, how are we today, man?\n",
            "Chandler: Hey, what are you doing?\n",
            "Joey: I don’t think this is so bad.\n",
            "Chandler: You know what? Whatever you are doing, just tell me. (To Joey) Tell it!\n",
            "Joey: Okay. (Hands him a cigar.)\n",
            "[Scene: Monica and Rachel's, Monica is sitting in the kitchen.]\n",
            "Monica: No, what are you doing?\n",
            "Chandler: I don’t know. I just wanted to have fun. And, who wants to have fun? I mean it’s not the sort of thing that’s been popular ever since we were kids.\n",
            "Monica: What do you want from her?\n",
            "Chandler: Oh-hah, just, like the other day, you wanted to go out. And then you had an argument. I just got out of the apartment and I called you. What, you know what? That way you couldn’t get out, but I’d always know. (Kicking his feet.) This morning I asked Chandler if it was cool to go out with her.\n",
            "Monica: Yeah, I heard that. Oh, what’s the matter, anyway? Is it the other way around?\n",
            "Chandler: No. No, I know what’s going on. I didn’t even know you were upset. Oh well, at least it’s over.\n",
            "Montage: Chandler & Joey go to bed. Chandler wakes up.\n",
            "Joey: I just woke up!\n",
            "Chandler: Oh, I had a dream that I was sleeping with her!\n",
            "Joey: (entering) I did it?!\n",
            "Chandler: It wasn’t that bad, but I didn’t make it past half past three.\n",
            "Joey: What?!\n",
            "Chandler: We talked about the same thing.\n",
            "Joey: I know, it-it’s not me that you’re upset about!\n",
            "Chandler: Well, it’s not me! I mean, I’ll see her on a very different level later.\n",
            "Monica: I'm gonna go get something to eat.\n",
            "Joey: Okay, I’ll get some coffee, okay? I’ll talk to you later. Okay, I’ll meet you up in ten or fifteen minutes.\n",
            "Chandler: Okay! Okay! (Points to the wall.) Okay! Alright! Here you go! Thanks for having dinner with me! (The camera pans up to where Joey is having dinner with Chandler.)\n",
            "Joey: Hey! (To Monica) I just think that what you were trying to show is that I’m a better person—I’m not.\n",
            "Chandler: Well, I know. That will help. (The camera pans back up to Joey. He has just finished a bowl of spaghetti.) That was a real bowl of spaghetti.\n",
            "[Scene: Monica and Rachel's, Ross is doing a voice over.]\n",
            "Ross: Hey, Chandler!\n",
            "Chandler: Ross, Ross!\n",
            "Ross: Chandler! Are you alright?\n",
            "Chandler: I’m fine. Just got a little cold hands today. Okay?\n",
            "Ross: Well, there’s been a lot of talk about spaghetti.\n",
            "Chandler: Yeah, I’m thinking about spaghetti again. (They start singing the song by the piano.) Okay.\n",
            "Ross: Oh, that’s better. That’s better. We can stop.\n",
            "Chandler: We can stop. Stop! We can stop! Stop! Stop! Stop! Stop! Stop! Stop (Chandler is singing.) Stop, stop, stop! Stop! Stop! Stop! Stop! Stop! Stop, stop! Stop! Stop! Stop, stop! Oh yeah. I’m sorry, we’re sorry Chandler, we’re sorry Chandler.\n",
            "Ross: All right. Well, at least I know that you are still upset.\n",
            "Chandler: Oh yeah. Okay, what-what just happened?\n",
            "Ross: Well, it was a dream!\n",
            "Chandler: You can just go to sleep?\n",
            "Ross: Yes. No, that is not right. It’s not okay!\n",
            "(The phone goes off.)\n",
            "Ross: Hello?\n",
            "(On speakerphone.) Hey.\n",
            "Closing Credits\n",
            "[Scene: Chandler and Joey's, Chandler is on the phone.]\n",
            "Chandler: (on phone) Hello. (The receiver hangs up.) Hello. (Chandler goes over and picks up the phone.) Hello, is this okay?\n",
            "\n",
            "[210 | 363.36] loss=2.18 avg=2.42\n",
            "[220 | 379.51] loss=2.66 avg=2.43\n",
            "[230 | 395.67] loss=2.10 avg=2.42\n",
            "[240 | 411.84] loss=2.66 avg=2.43\n",
            "[250 | 428.02] loss=2.23 avg=2.42\n",
            "[260 | 444.22] loss=2.26 avg=2.41\n",
            "[270 | 460.41] loss=2.83 avg=2.43\n",
            "[280 | 476.62] loss=2.48 avg=2.43\n",
            "[290 | 492.85] loss=1.93 avg=2.41\n",
            "[300 | 509.08] loss=2.26 avg=2.41\n",
            "[310 | 525.32] loss=2.34 avg=2.40\n",
            "[320 | 541.55] loss=2.22 avg=2.40\n",
            "[330 | 557.79] loss=2.20 avg=2.39\n",
            "[340 | 574.04] loss=2.34 avg=2.39\n",
            "[350 | 590.28] loss=2.51 avg=2.39\n",
            "[360 | 606.52] loss=2.52 avg=2.40\n",
            "[370 | 622.76] loss=2.25 avg=2.39\n",
            "[380 | 639.01] loss=2.52 avg=2.40\n",
            "[390 | 655.26] loss=2.21 avg=2.39\n",
            "[400 | 671.52] loss=2.52 avg=2.39\n",
            "======== SAMPLE 1 ========\n",
            ":\n",
            "[Scene: At Monica and Rachel's Apartment, everyone is there at Betty & Veronica's, with Phoebe, Chandler, and Joey having dinner on the sofa.]\n",
            "Joey: This is awful. I'm such a man. (He leans in slightly to Monica, who is still speaking Spanish, and the two of them stare into each other's eyes.)\n",
            "[Cut to Monica as Rachel comes over, leaving Betty and Veronica alone at the kitchen table.]\n",
            "Betty: Hi! Have a great dinner!\n",
            "Verdica: And the best part: (Chandler and Joey look at her.)\n",
            "Chandler: And also, I love your shoes! (Chandler and Joey look at each other for a few seconds, and they nod.)\n",
            "Nurse: Oh, yeah, you did a fine job. (Nurse leans in towards the door.) Hey. (Rachel is having sex with her best man with who's penis is slightly visible on the other side of her mouth. She leans in closer and when he pushes her in a little too fast, there comes the sound of her mouth opening.) Hey, it's me! Hey, hey, hey, (Monica gasps) That was amazing! Yes, it was amazing, I'm totally gonna get married this summer! I'll get it! I'll get me that ring. Well, I'm a little late for my honey and I'm a little embarrassed about it! (Monica stares at her, but not for long.)\n",
            "Chandler: (To her) I-I think I saw what you did.\n",
            "Joey: Oh yeah, yes! Me too! I saw what you did!\n",
            "Chandler: That was wonderful! I see what you've done. I see your point, I'm gonna talk to Ross about this! Okay fine. (Rachel goes back to talking to Monica.)\n",
            "Rachel: Yeah, sure!\n",
            "Nurse: Your wife is going to the doctor!\n",
            "Chandler: No, she's fine.\n",
            "Nurse: I don't think it's a big deal.\n",
            "Joey: Okay, I can see how important it is, and I'll be happy.\n",
            "Chandler: Oh, I'm sure her first time will be really strange, I mean, I can't even believe I didn't tell her.\n",
            "Nurse: (To Joey) She's a lovely woman!\n",
            "Joey: Ahh, I am not? (Chandler gasps.)\n",
            "[Scene: Ross's Apartment, they're all there as Joey is sitting on the couch as the news reporter comes knocking and they all answer awkwardly.]\n",
            "Joey: Uh, hey, hi.\n",
            "Chandler: Hey, Ross.\n",
            "Joey: Hey, hi! Hi! Hey, that's Ross.\n",
            "Chandler: How's the baby?\n",
            "Ross: Hi, y'know, when the news reporter called I said I'm gonna be here! I've got no chance at all! I've got a plan!\n",
            "Chandler: Why?\n",
            "Ross: I can't do it if you don't know about it!\n",
            "Chandler: Okay, well, what's the problem? I don't think you know.\n",
            "Ross: Because, you know I don't think about it, and that's why everybody's so upset. And the only way I know anyone's gonna be upset is if somebody I'm dating breaks in and does something stupid. And they won't be mad at me because they've already made all this shit up about how, how stupid you are! (Everyone looks at him, and he's shocked again.)\n",
            "Chandler: What?! You did the same thing that got you fired from that job?\n",
            "Ross: Yeah. Yeah. Yeah.\n",
            "Chandler: That's good in this job! I mean, it's definitely gonna make someone want you! All right? I mean, I'd really like to get married! And if I were you I would want to get married! (Chandler is shocked again.)\n",
            "Phoebe: Oh what?\n",
            "Rachel: Oh hey, you know what you can do? I do. Okay? I'm gonna go out with you and I'll be out by Friday, but uh! (They're talking in a passionate language which sounds quite a bit like the English spoken around them.) Hey, listen, I'm going out with (insert name of the guy), and I'm sorry. So, I'm going out to dinner as a friend, and (pointing to the guy) I'm married.\n",
            "Phoebe: Oh, no we don't! We don't!\n",
            "Rachel: Hey, you think? Yeah? Just don't marry him to a girl who wouldn't be married to you?!\n",
            "Phoebe: Okay.\n",
            "Joey: I think you're crazy. Okay?\n",
            "(There's a knock\n",
            "\n",
            "[410 | 709.58] loss=2.14 avg=2.39\n",
            "[420 | 725.85] loss=2.29 avg=2.38\n",
            "[430 | 742.11] loss=2.13 avg=2.38\n",
            "[440 | 758.38] loss=2.44 avg=2.38\n",
            "[450 | 774.66] loss=2.42 avg=2.38\n",
            "[460 | 790.92] loss=2.57 avg=2.38\n",
            "[470 | 807.17] loss=2.41 avg=2.39\n",
            "[480 | 823.44] loss=2.25 avg=2.38\n",
            "[490 | 839.70] loss=2.52 avg=2.39\n",
            "[500 | 855.95] loss=2.12 avg=2.38\n",
            "Saving checkpoint/run1/model-500\n",
            "[510 | 883.83] loss=2.01 avg=2.37\n",
            "[520 | 899.85] loss=2.49 avg=2.37\n",
            "[530 | 915.95] loss=2.04 avg=2.36\n",
            "[540 | 932.06] loss=1.81 avg=2.35\n",
            "[550 | 948.23] loss=2.28 avg=2.35\n",
            "[560 | 964.46] loss=2.23 avg=2.35\n",
            "[570 | 980.73] loss=2.15 avg=2.34\n",
            "[580 | 997.03] loss=2.28 avg=2.34\n",
            "[590 | 1013.33] loss=2.16 avg=2.34\n",
            "[600 | 1029.62] loss=2.42 avg=2.34\n",
            "======== SAMPLE 1 ========\n",
            " leaves you wondering, why you did it… Did you know, I’m-I’m gonna do it again?\n",
            "Rachel: No, I wasn’t, I didn’t do it again, okay?\n",
            "Ross: But, if you have to do that, you’re gonna have to. I… I… (They walk away from each other.)\n",
            "[Scene: Monica and Rachel's, Chandler and Monica are inside.]\n",
            "Chandler: I know what you’re really doing. You’re probably just messing up your feelings here.\n",
            "Monica: (to Rachel) You’re right. I think that is what you’re doing.\n",
            "Chandler: I know. But you’re right, if, (to himself, and to Monica) I’m messin’ this up, I’m gonna have to stop it.\n",
            "Monica: You know, you’re right. You’re right, I’m sorry. It’s just that there’s this huge weight on your shoulder.\n",
            "Chandler: I know. Well, I just-I just have to see you again. I can’t help thinking–\n",
            "Monica: Sure. You do.\n",
            "Chandler: We’re just two people who like to love each other, whose lives are in complete harmony, and who’re just in love with each other.\n",
            "Monica: All right, what are we doin’?\n",
            "Chandler: Let me start at the beginning.\n",
            "Monica: Okay.\n",
            "Chandler: Yeah, I… I think that we need to get back together. Okay.\n",
            "Monica: What?!\n",
            "Chandler: I don’t want to get back together with Rachel. Because I said I would. Okay? And I thought, if I get this far, we’g-we’d have to start all over.\n",
            "Monica: Okay, I’m afraid that the end of this moment, is really not that far off.\n",
            "Chandler: I understand. I still have a lot of feelings about Rachel and that’s why we were going to start all over.\n",
            "Monica: Okay!\n",
            "Chandler: There was a part of me which thought that maybe I’m just gonna make it happen and maybe we’re really not that far, I just think that… I mean it’s not that hard?\n",
            "Monica: Chandler I can’t do this!\n",
            "Chandler: No, it’s not that hard.\n",
            "Monica: Come on man! I don’t believe it! This is my fault! I mean just because I got to start all over again.\n",
            "Chandler: No, no!\n",
            "Monica: You’re right! You’re right!\n",
            "Chandler: Yeah. I’m so sorry. I don’t feel like I had like a choice. I just, I just have to say to Monica that I like you better!\n",
            "Monica: I like you better than you.\n",
            "Chandler: No, I’m not like you! No! No. No. Okay.\n",
            "Monica: I’m sorry! I said no!\n",
            "Chandler: All right okay, okay. I’ll just do that one.\n",
            "Monica: Okay. Oh, please.\n",
            "Chandler: And, Monica?\n",
            "Monica: No. Can I go and talk to you?\n",
            "Chandler: Yeah.\n",
            "Monica: No no, we could sit down.\n",
            "Chandler: Do you hear what I’m thinking?\n",
            "Monica: Yeah.\n",
            "Chandler: I don’t know if Joey’s there, but it’s not like he hates you. That’s the worst thing you can do in this situation. The worst thing you can do is think that maybe you’re not like me, you’re a wonderful person, you’re a talented musician. (Pause) Well… Okay! I have to do this! I’ve done it before! And I’ve done it when, and I thought you were better! You’re just, you’re meaner and you’re meaner! And your mother wouldn’t have said, \"You’re meaner, and meaner and meaner!\" Now, if you’re meaner, you’re meaner.\n",
            "[Scene: Central Perk, Ross is walking in and Rachel is there.]\n",
            "Ross: So Rachel! You said that you liked me, and I told you I wasn’t interested.\n",
            "Rachel\n",
            "\n",
            "[610 | 1067.57] loss=1.95 avg=2.33\n",
            "[620 | 1083.89] loss=2.33 avg=2.33\n",
            "[630 | 1100.19] loss=1.97 avg=2.32\n",
            "[640 | 1116.49] loss=2.03 avg=2.32\n",
            "[650 | 1132.79] loss=2.36 avg=2.32\n",
            "[660 | 1149.09] loss=2.53 avg=2.32\n",
            "[670 | 1165.39] loss=1.92 avg=2.31\n",
            "[680 | 1181.68] loss=2.42 avg=2.32\n",
            "[690 | 1197.98] loss=2.27 avg=2.31\n",
            "[700 | 1214.26] loss=2.10 avg=2.31\n",
            "[710 | 1230.56] loss=2.40 avg=2.31\n",
            "[720 | 1246.86] loss=2.28 avg=2.31\n",
            "[730 | 1263.15] loss=2.15 avg=2.31\n",
            "[740 | 1279.46] loss=2.52 avg=2.31\n",
            "[750 | 1295.78] loss=2.11 avg=2.31\n",
            "[760 | 1312.08] loss=1.96 avg=2.30\n",
            "[770 | 1328.40] loss=2.23 avg=2.30\n",
            "[780 | 1344.70] loss=2.57 avg=2.31\n",
            "[790 | 1361.02] loss=2.47 avg=2.31\n",
            "[800 | 1377.31] loss=2.68 avg=2.32\n",
            "======== SAMPLE 1 ========\n",
            " you you're just getting off of work.\n",
            "Rachel: (interupting him again) What?\n",
            "Monica: What?\n",
            "Rachel: Oh my God it was a duck and it was in your room. Why did you do that?\n",
            "Ross: Yeah. I mean it was a duck. I was trying to get her to like me.\n",
            "Rachel: Oh my God! So you don't want to live with her!\n",
            "Ross: Yeah, you don't want to live with her. I mean, I'm so sorry. (He turns to the other couple.)\n",
            "Rachel: Ok, sorry you're not going to have her.\n",
            "Ross: You aren't. I mean if there was one thing you should know I know it was that.\n",
            "(The phone rings and Monica answers it.)\n",
            "Monica: Hi.\n",
            "Rachel: Oh no-no, Ross?\n",
            "Ross: Yeah?\n",
            "[Scene: Monica, Chandler, and Joey's, Joey is writing on an easel in a red pen and Chandler comes out of his room carrying some tissues.]\n",
            "Chandler: Yay!\n",
            "Joey: Whoo!\n",
            "Chandler: Hey, what are you doing here?!\n",
            "Joey: You're gonna write in my room.\n",
            "Chandler: (entering) Here. (He moves the pen to the paper and Chandler looks shocked.) Hey, what are you doing here?\n",
            "Monica: I didn't know that I could get you out of here.\n",
            "Chandler: Uh, what're you doing here? Do you have an audition for me?\n",
            "Monica: No. Why? It's nothing, I just, I was um, I was in a dark part of the building. (He laughs.) I'm just checking, uh, just checking you out and, um, you'd be right at home.\n",
            "Chandler: Oh, yes I do. You need a place to rehearse it?\n",
            "Monica: Sure, and why, why not, uh, if I don't get it, will you give me a call or whatever the answer is.\n",
            "Chandler: I'm on the phone. It's Monica. Just call.\n",
            "Monica: Do you need an audition right now?\n",
            "Chandler: Yeah?\n",
            "Monica: Yeah.\n",
            "Chandler: So how come you have my audition so easy?\n",
            "Monica: All right, look, y'know what, if I get it…\n",
            "Chandler: Uh honey, I've got a deal here. I promise.\n",
            "Monica: How much money do you have in your bank account?\n",
            "Chandler: Nothing.\n",
            "Monica: You have a girlfriend, eh? Don't you feel like you've been flirting a little too much?\n",
            "[Scene: Monica, Chandler, and Joey's, Ross and Angela are sitting on the couch and talking.]\n",
            "Angela: How much is it now?\n",
            "Ross: About a million.\n",
            "Angela: (treading her feet) Oh my God. Oh, wow.\n",
            "Ross: Look, I haven't said this to a woman in a long time! You should meet a man and date him, so that he may get your interest.\n",
            "(The phone rings and Rachel enters to find Monica with the envelope containing Chandler's audition, Phoebe enters carrying a book.]\n",
            "Phoebe: Hey.\n",
            "Rachel: (sees the paper): (in a high pitched voice) What an audition for a man in a cab? (Miming a cab driver.) \"Hey, you look hot.\" (Miming the cab driver.) \"Hey, how's your back?\" I mean this is like, this is like the, this is like the thing I've always wanted to do with my life. How do I explain how I did it I mean? I mean it is so hard but…\n",
            "Phoebe: I could never do it just once.\n",
            "Rachel: Oh, I could never do it…\n",
            "Phoebe: No Rachel, never.\n",
            "(Phoebe puts the script back and starts to cry.)\n",
            "Monica: Do you have one?\n",
            "Rachel: Oh no.\n",
            "Phoebe: Why not?\n",
            "Rachel: Well, there's this woman... (Coughs.)\n",
            "Phoebe: I mean, it's like this woman that you feel really close to. I mean you just, you just… (Giggles, to Chandler and Monica) ...I mean, she… (Starts to cry)\n",
            "(Joey comes out of Monica and Rachel's as Rachel is writing and starts to cry.)\n",
            "Monica: How was your audition?\n",
            "Joey: Great! I was just here to see if they would accept this. (Waves) Wow, you guys are really good.\n",
            "Monica: That's great! You really can do that.\n",
            "(Joey\n",
            "\n",
            "[810 | 1415.87] loss=1.86 avg=2.31\n",
            "[820 | 1432.18] loss=2.58 avg=2.31\n",
            "[830 | 1448.47] loss=2.36 avg=2.31\n",
            "[840 | 1464.76] loss=2.19 avg=2.31\n",
            "[850 | 1481.06] loss=1.71 avg=2.30\n",
            "[860 | 1497.34] loss=2.20 avg=2.30\n",
            "[870 | 1513.63] loss=2.21 avg=2.30\n",
            "[880 | 1529.92] loss=2.25 avg=2.30\n",
            "[890 | 1546.22] loss=2.30 avg=2.30\n",
            "[900 | 1562.52] loss=1.85 avg=2.29\n",
            "[910 | 1578.82] loss=2.39 avg=2.29\n",
            "[920 | 1595.12] loss=2.26 avg=2.29\n",
            "[930 | 1611.42] loss=2.22 avg=2.29\n",
            "[940 | 1627.73] loss=1.96 avg=2.28\n",
            "[950 | 1644.03] loss=2.15 avg=2.28\n",
            "[960 | 1660.32] loss=2.16 avg=2.28\n",
            "[970 | 1676.62] loss=1.90 avg=2.27\n",
            "[980 | 1692.93] loss=2.45 avg=2.28\n",
            "[990 | 1709.23] loss=1.86 avg=2.27\n",
            "[1000 | 1725.52] loss=2.04 avg=2.27\n",
            "Saving checkpoint/run1/model-1000\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to delete files with this prefix.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IXSuTNERaw6K",
        "colab_type": "text"
      },
      "source": [
        "模型训练完之后，你就可以把保存模型，直接拷到 Google Drive 下面去了。\n",
        "\n",
        "\n",
        "之后如果想在自己本地使用的话，就直接从 Google Drive 直接下载就可以了。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VHdTL8NDbAh3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gpt2.copy_checkpoint_to_gdrive()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qQJgV_b4bmzd",
        "colab_type": "text"
      },
      "source": [
        "OK，搞定了。\n",
        "\n",
        "现在就尽情调戏自己训练好的模型吧。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pel-uBULXO2L",
        "colab_type": "text"
      },
      "source": [
        "## 载入训练好的模型\n",
        "\n",
        "下面的 cell 会把你 google drive 下的 `checkpoint` 拷进来。\n",
        "\n",
        "如果是一口气直接训练过来的话，就不用运行这一步了。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DCcx5u7sbPTD",
        "colab_type": "code",
        "outputId": "3c8c4bc0-3295-4d3a-fc85-b2e6a3116322",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "gpt2.copy_checkpoint_from_gdrive()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "FileExistsError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileExistsError\u001b[0m                           Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-3-ea9a7bfc97cc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mgpt2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy_checkpoint_from_gdrive\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/gpt_2_simple/gpt_2.py\u001b[0m in \u001b[0;36mcopy_checkpoint_from_gdrive\u001b[0;34m(checkpoint_folder)\u001b[0m\n\u001b[1;32m    460\u001b[0m     \u001b[0mis_mounted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    461\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 462\u001b[0;31m     \u001b[0mshutil\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopytree\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"/content/drive/My Drive/\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mcheckpoint_folder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcheckpoint_folder\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    463\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    464\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/shutil.py\u001b[0m in \u001b[0;36mcopytree\u001b[0;34m(src, dst, symlinks, ignore, copy_function, ignore_dangling_symlinks)\u001b[0m\n\u001b[1;32m    313\u001b[0m         \u001b[0mignored_names\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    314\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 315\u001b[0;31m     \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmakedirs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdst\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    316\u001b[0m     \u001b[0merrors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    317\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnames\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/os.py\u001b[0m in \u001b[0;36mmakedirs\u001b[0;34m(name, mode, exist_ok)\u001b[0m\n\u001b[1;32m    218\u001b[0m             \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    219\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 220\u001b[0;31m         \u001b[0mmkdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    221\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    222\u001b[0m         \u001b[0;31m# Cannot rely on checking for EEXIST, since the operating system\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mFileExistsError\u001b[0m: [Errno 17] File exists: 'checkpoint/run1'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTa6zf3e_9gV",
        "colab_type": "text"
      },
      "source": [
        "之后就是，从模型中载入训练好的模型。\n",
        "\n",
        "这个 cell 和之前训练 cell 一样，如果想重复跑的话得重启环境。\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-fxL77nvAMAX",
        "colab_type": "code",
        "outputId": "38ca8c2e-64d6-470b-a374-eaf927bb3ef8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 171
        }
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "gpt2.load_gpt2(sess)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "Loading checkpoint checkpoint/run1/model-1000\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n",
            "INFO:tensorflow:Restoring parameters from checkpoint/run1/model-1000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClJwpF_ACONp",
        "colab_type": "text"
      },
      "source": [
        "## 从训练好模型生成文本\n",
        "\n",
        "载入模型之后，就是激动人心的生成部分了！\n",
        "\n",
        "下面这个是无条件（Unconditional）的生成，也就是没有任何限制，让模型自己生成。\n",
        "\n",
        "结果并不理想，可能因为 finetune 的步数太小了，还有一些生成参数没有设置，这里只是个示例所以就只设 1000 步训练，想要更好效果可以设更大一些。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4RNY6RBI9LmL",
        "colab_type": "code",
        "outputId": "f4f69413-981e-48ab-c7f6-3ce62e6a45a5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1448
        }
      },
      "source": [
        "gpt2.generate(sess)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:51: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:53: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.random.categorical instead.\n",
            "Actress - The One With The Wedding Dress\n",
            "\n",
            "Ally: (entering) Hey!\n",
            "Ross: Hi!\n",
            "Rachel: Hi!\n",
            "Phoebe: Hey! (they hug)\n",
            "Ross: Hi!\n",
            "Rachel: Hi! I didn't know you were getting married!\n",
            "Phoebe: You didn't know.\n",
            "Ross: I didn't know!\n",
            "Rachel: I didn't know but y'know, I have to get married!\n",
            "Phoebe: Oh, I just...\n",
            "Rachel: I don't know.\n",
            "Phoebe: Okay, well then, I have to get married.\n",
            "Rachel: Oh, oh, I can't wait to get married! You know, if you don't know I'm gonna talk to you. I mean, I know that you're gay, but... I mean, you're gay and-and you're married.\n",
            "Phoebe: Oh, I'm so sorry, but y'know I just have to go to the wedding.\n",
            "Rachel: Oh no, no, no, you don't have to go to the wedding.\n",
            "Phoebe: No, I'll see you tomorrow.\n",
            "Rachel: Okay. (to Ross) Good for you!\n",
            "Ross: Great. Thank you!\n",
            "Rachel: Thank you!\n",
            "Ross: Okay. (to Phoebe) No, no, I'm not gay. I'm, I'm just, I'm just going to see you tomorrow.\n",
            "Phoebe: Oh, yeah, but, y'know I'm gonna see you too!\n",
            "Ross: You are going to see me too.\n",
            "Phoebe: I'm going to see you too.\n",
            "Ross: You are going to see me too!\n",
            "Phoebe: I'm going to see you too.\n",
            "Ross: Okay. (to Ross) Well, would you go to the wedding with me?\n",
            "Ross: Very cute.\n",
            "Phoebe: You're gonna see me too.\n",
            "Rachel: (entering) Hey Ross!\n",
            "Joey: Hey!\n",
            "Rachel: Hey!\n",
            "Ross: Hi!\n",
            "Phoebe: Hi!\n",
            "Rachel: Hey!\n",
            "Ross: Hi!\n",
            "Phoebe: Hi!\n",
            "Rachel: I, I just got off the phone with Joey.\n",
            "Ross: What's up?\n",
            "Phoebe: Oh, nothing, nothing, nothing. Just uh, just that our, our plans are changing and y'know, we're going to see each other next week, but we haven't talked about which way.\n",
            "Ross: Yeah, but, but, y'know Joey you're gonna see me too.\n",
            "Phoebe: Yes, Joey is gonna see me too!\n",
            "Rachel: I'm so sorry!\n",
            "Ross: No, it's not, it's not.\n",
            "Phoebe: No, I was so worried that I would see you both way.\n",
            "Rachel: No, no, I'm not, I'm not, I'm not worried.\n",
            "Ross: No! No, I'm definitely not worried.\n",
            "Phoebe: No, you're not worried, you're not worried!\n",
            "Rachel: No, \"I'm not worried\" on TV is not a word, it is not!\n",
            "Ross: Yeah, I mean, there are all these countries that give you free tickets to see each other, but I mean, you don't have to go, y'know, to the wedding.\n",
            "Phoebe: No, it's just... It's just, y'know, y'know, it's just not a way that I'd be able to... Wait, Ross, y'know, I'm telling you, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried.\n",
            "Rachel: No, I'm not worried.\n",
            "Ross: No, I'm not worried\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oF4-PqF0Fl7R",
        "colab_type": "text"
      },
      "source": [
        "之后如果你想要创建 API 获得生成的文本的话，直接运行 `text = gpt2.generate(sess, return_as_list=True)[0]` 就好了。\n",
        "\n",
        "然后，上面提到了无条件生成，那有条件生成(Conditional)呢，我们可以直接用`prefix`来指定一段文本，然后让模型来续写。\n",
        "\n",
        "此外，还可以同时生成几个样本，通过指定`nsamples`。还有，通过设定`batch_size`还可以加速生成过程。\n",
        "\n",
        "\n",
        "`gpt2.generate` 和相关函数其他一些有用的选项：\n",
        "\n",
        "*  **`length`**: 生成文本长度 (默认 1023, 也是可设最大长度)\n",
        "* **`temperature`**: temperature 越高，生成的就随意。 (默认 0.7，推荐 0.7 到 1.0之间)\n",
        "* **`top_k`**: 将输出限定在 top k 里面 (默认0，也就是不使用。推荐在生成效果差的时候使用，可以设top_k=40)\n",
        "* **`truncate`**: 从指定符号阶段生成文本 (比如设 `truncate='<|endoftext|>'`, 那么就会取第一个'<|endoftext|>'前的文本作为输出). 可以和一个比较小的`length`值搭配使用.\n",
        "*  **`include_prefix`**: 如果用了 `truncate` 和 `include_prefix=False`, 那么在返回文本中就不会包含`prefix`里的文本。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DKMc0fiej4N",
        "colab_type": "code",
        "outputId": "3263a7e0-2fcf-45b8-f5ce-885a71136457",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1599
        }
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              length=250,\n",
        "              temperature=0.7,\n",
        "              prefix=\"Rachel: Andy, marry me!\",\n",
        "              nsamples=5,\n",
        "              batch_size=5,\n",
        "              top_k=40\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "R: Andy, marry me!\n",
            "Andy: (hugs her) Well, it's a good one, no.\n",
            "Rachel: Oh-oh, I’m sorry.\n",
            "Andy: I’ve just got to get a new apartment.\n",
            "Rachel: Oh no, no-no!!\n",
            "Andy: No-no, no. No, no-no.\n",
            "Rachel: No-no-no.\n",
            "Andy: No, no, no-no-no. (Moves Rachel to the door.) (opens it) You’re not moving.\n",
            "Rachel: No-no-no-no, not moving. No!!\n",
            "Andy: Oh, come on! You’re not moving!\n",
            "Rachel: No, not moving. No!!\n",
            "Andy: No-no, not moving. No!!\n",
            "Rachel: No-no, not moving. No!!\n",
            "Andy: No-no, not moving. No!!\n",
            "Rachel: No, not moving!!\n",
            "Andy: No-no, not moving. No!!\n",
            "Rachel: No-no!\n",
            "Andy: No-no-no!!\n",
            "Rachel: (closes the door and runs out on her own, slamming the door behind her) I’m\n",
            "====================\n",
            "\n",
            "R: Andy, marry me!\n",
            "Joey: No! No! No!\n",
            "Chandler: You can't do this!\n",
            "Rachel: We have to do something!\n",
            "Joey: No! You can't!\n",
            "Chandler: Yes you can!\n",
            "Joey: Yes you can!\n",
            "Chandler: I can!\n",
            "Joey: Yeah! (He pushes Chandler back onto his bed.)\n",
            "Chandler: I can!\n",
            "Joey: Okay! Okay! Okay! Okay- Okay!\n",
            "Chandler: Okay! Okay!\n",
            "Joey: Okay! Okay! (He gets up to get up.) Okay! (He gets up).\n",
            "Joey: Okay! (He gets up.) Okay! Okay! (He gets up to get up.) Okay! Okay! (He gets up to get up.) Okay!\n",
            "Chandler: Yeah!\n",
            "Joey: Okay!\n",
            "Chandler: Okay!\n",
            "Joey: Okay! (He gets up to get up.) Okay! Okay! Okay! (He gets up to get up.) Okay! (He gets up to get up.) Okay! Okay! Okay! Okay! Okay! Okay! Okay! Okay! Okay! Okay! Okay\n",
            "====================\n",
            "\n",
            "R: Andy, marry me!\n",
            "Andy: Oh, no, I'm sorry.\n",
            "Rachel: Oh, come on…\n",
            "Andy: I'm sorry.\n",
            "Rachel: Oh-oh-hey Andy, I'm sorry.\n",
            "Andy: I'm sorry.\n",
            "Rachel: Oh, I'm sorry. I'm so sorry. (She hugs him and kisses him on the lips)\n",
            "Andy: Thank you. (She hugs him again)\n",
            "Rachel: Oh, sorry, I'm just so sorry.\n",
            "Andy: Ok, so you know what I just say?\n",
            "Rachel: Oh, no-no-no, not even?\n",
            "Andy: You do know what I just say?\n",
            "Rachel: No-no-no I'm sorry, I'm so sorry, I'm so sorry.\n",
            "Andy: Ok, so you know what I'm gonna do?\n",
            "Rachel: No, actually I'm just, I'm gonna say this, I'm so sorry. I'm so sorry. I'm so sorry. (she hugs him again)\n",
            "Andy: I'm so sorry.\n",
            "Rachel: Ok, so you know what I'm gonna say?\n",
            "Andy: You are going to kiss me.\n",
            "Rachel: What?\n",
            "Andy: You are\n",
            "====================\n",
            "\n",
            "R: Andy, marry me! Come on!\n",
            "Andy: (suddenly) I don't know what I'm gonna do.\n",
            "Ross: (to Rachel) Now, I'm only going to marry one of you.\n",
            "Rachel: No, I mean, I mean, you really think I'm gonna break up with you?\n",
            "Ross: So, I do. (He kisses her.) And I love you.\n",
            "The End\n",
            "408 The One Where Ross Finds Out\n",
            "[Scene: Monica and Chandler's, Monica is sitting on the couch, Chandler is writing with his pen, and Chandler is watching. Monica is trying to make the phone ring.]\n",
            "Monica: (to Chandler) Well, that phone didn't answer.\n",
            "Chandler: No, I'm on the phone.\n",
            "Monica: (on phone) Hello? (Listens) Uh, it's going to be a long day. (Listens) I'm sorry you're late. (Listens) It's going to be a long day. (Listens) (To Chandler) I'll be right back. (Listens) (To Monica) I'll be right back. (Pause) (To Chandler) I'll be right back. (Listens\n",
            "====================\n",
            "\n",
            "R: Andy, marry me!\n",
            "Andy: Nooo, I guess that's...\n",
            "Monica: What?\n",
            "Andy: I said: (To the girls) I want to see you again. (To Monica) I... (Monica starts to cry)\n",
            "Joey: (To Chandler) You're so cute and you know so much about the business world!\n",
            "Rachel: Thank you. (To Monica) Oh, it's so great to see you guys again, y'know? I mean, you guys are so amazing and you know so much about the business world.\n",
            "Joey: Yeah.\n",
            "Rachel: Y'know, I mean, how long do you think I'll be able to see you again?\n",
            "Joey: I don't know, y'know? Maybe like forever.\n",
            "Rachel: No, no, just noo!\n",
            "Monica: Wait, wait, wait, wait, wait!\n",
            "Rachel: No.\n",
            "Monica: So, you wanna do it?\n",
            "Rachel: How long do you think it'll be?\n",
            "Joey: Oh, y'know what? I'm gonna do it. Look, I'm gonna be with you guys forever and I'm so, so happy for you\n",
            "====================\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjjEN2Tafhl2",
        "colab_type": "text"
      },
      "source": [
        "如果想要生成大量文本的话，就可以用下面 cell 的命令。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fa6p6arifSL0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())\n",
        "\n",
        "gpt2.generate_to_file(sess,\n",
        "                      destination_path=gen_file,\n",
        "                      length=500,\n",
        "                      temperature=0.7,\n",
        "                      nsamples=100,\n",
        "                      batch_size=20\n",
        "                      )\n",
        "\n",
        "files.download(gen_file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ig-KVgkCDCKD",
        "colab_type": "text"
      },
      "source": [
        "# 额外提示\n",
        "\n",
        "如果出现错误 (比如 GPU Sync Fail or out-of-memory/OOM), 可以用下列命令来强行杀掉并重启:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rIHiVP53FnsX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!kill -9 -1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmTXWNUygS5E",
        "colab_type": "text"
      },
      "source": [
        "欢迎转载，务必注明出处。\n",
        "\n",
        "\n",
        "关注公众号：Andy的写作间(andy_writing) ，更多自然语言处理，深度学习，AI，写作等等等"
      ]
    }
  ]
}